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The  Sobering  State  of  Software  Engineering 


39% 

43% 

18% 


of  software  projects  are 


successful 


of  software  projects  cost  more, 

take  longer,  or  do  less 


of  software  projects 


failed 


The  2013  Chaos  Manifesto  -  The  Standish  Group  -  http://versionone.eom/assets/img/files/CHAOSManifesto2013.pdf 
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Another  Data  Point 


In  a  survey  of  166  IT  leaders: 


89% 


of  projects  do  not  regularly  meet  their  budget 


59% 


projects  are  typically  delivered  late 


33% 


state  that  rework  is  at  least  25%  of  their  budget 


2014  IT  Leadership  Survey  -  Blueprint  Software  Systems  Inc. 
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Can  Measurement  Help? 


CEO’s  have  a  lower  opinion  of  software  groups  than  of  other 
technical  groups  due  to  consistently  optimistic  estimates,  schedule 
delays,  cost  overruns,  poor  quality  when  delivered,  and  outright 
failures.  Software  is  much  worse  in  all  of  these. 


Better  measures  of  projects  ...  will  improve  the 
professional  status  of  the  software  community  and  perhaps 
lead  to  CEO’s  having  more  respect  for  software  groups 
than  they  have  today. 


finger 


Capers  Jones 
InfoQ  Interview 
March  30,  2014 

http://www.infoq.com/articles/Jones-measuring-agile-adoption 
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The  World  Without  Measurement 


\ 

Medicine? 


— ^  When  Measurement  Benefits  the  Measured 

Software  Engineering  Institute  Carnegie  Mellon  University  Kasunics  Nichols,  April  23, 2014  ^ 

©  2014  Carnegie  Mellon  University 


We  All  Measure  .... 


Cooking?  Getting  to  Work  on  Time? 
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How  We  Use  Measurement 

First-Order  Measurement 

What  seems  to  be  happening? 

Tends  to  be  qualitative  and  fast. 

Second-Order  Measurement 

What’s  really  happening?  And  how  is  it  changing? 

It  needs  to  be  quantitative;  subject  to  more  refined  models. 

Third-Order  Measurement 

What  happens  in  a  more  general  and  universal  sense? 

Needs  to  be  precise  with  checks  for  validity;  statistical  variation  must 
be  characterized  and  interpreted  appropriately. 


Gerald  (Jerry)  Weinberg 


— ^  When  Measurement  Benefits  the  Measured 

Software  Engineering  Institute  Carnegie  Mellon  University  Kasunics  Nichols,  April  23, 2014  9 

©  2014  Carnegie  Mellon  University 


Third-Order  Measurement 


Software  Engineering  Institute 


Carnegie  Mellon  University 
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The  State  of  the  Practice 


An  Issue 

The  results  of  applying  many  software  development  methods  are  unpredictable. 

Decision  making  about  method  selection  is  based  on  suppositions,  opinions,  and 
fads. 


What  We  Need 


We  need  to  set  aside  perceptions  and  market-speak  . . .  and  transform  software 
engineering  into  an  engineering  discipline. 


Decisions  should  be  based  on  fair  and 
unbiased  analysis  of  information. 


Quality 


i 


Scope 
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Measurement  in  Your  Work 


The  Team 


Individual 
Software  Engineer 


The  Organization 


Software  Engineering  Institute  Carnegie  Mellon  University 
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Measurement  &  the  Individual  Software  Engineer 


O 


You 


Software  Engineering  Institute 


Carnegie  Mellon  University 
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stellar  athletes  understand 
that  they  must  set  specific 
goals  to  reach  their 
potential. 


Measurement  provides  the 
necessary  feedback  that  drives 
athletes  to  achieve  world-class 
excellence. 
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Can  software  engineers 
leverage  goal-setting 
and  measurement  in  the 
same  way? 
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Software  Engineering  Institute 


Car 


Do  you? 
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What  type  of  measures  do  you  typically  use  to  estimate  the  duration  of 
your  work  for  a  schedule? 


□ 

□ 

□ 

□ 


First  order:  Qualitative  -  based  on  what  I  think  I’ve  done  before. 

Second  order:  Quantitative  -  based  on  quantitative  data  from 

previous  project(s). 

Third  order:  Statistical  -  based  on  statistical  patterns  of  data  from 

my  previous  projects. 

None  of  the  above. 


□ 

□ 

□ 

□ 


Should  You  Be  Using  Measurement? 


Yes.  And  it  needs  to  go  beyond 
first  order  measurement. 

Measurement  is  needed  to 

manage  your  work. 
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Managing  the  Work 


But  isn’t  it  the 
managers  job  to 
manage? 
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The  Evolution  of  the  Management  Approach 


Body  Management 

People  as  oxen. 


Frederick  Taylor 


Peter  Drucker 


Task  Management  Knowledge  Management 

People  as  machines.  People  as  individuals. 
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Evolution  of  the  Worker 


Hunter-Gatherer 


Farmer  &  Artisan 


Industrial  Revolution  Worker 


Technoiogy  Professional 
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Taylorism  -  Scientific  Management 


For  years,  the  basic  power  equation  in  organizations  was  simple  and 
effective: 

Knowledge  held  by  a  few  (the  managers),  plus  iron  discipline  over  the 
many  (the  workers). 


The  worker  was  viewed  as  an  instrument,  a  bundle  of  muscles 
programmed  through  instruction. 
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The  Birth  of  the  Knowledge  Worker 


The  Technology  Professional 


New  data  processing  age  was 
born  during  2^^  half  of  20^^ 
century 

Work  became  asynchronous  and 
non-linear 

Nature  of  knowledge  work 
demanded  significant  control  by 
the  worker  (instead  of  the 
manager) 
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What  Differentiates  Knowledge  Work? 


Manual  work 

Consists  of  converting  materials  form 
one  form  to  another. 

The  work  output  is  tangible. 


Knowledge  work 

The  work  is  done  in  the  head. 
The  work  can’t  be  seen. 
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A  Shift  In  the  Locus  of  Control 


This  new  breed  of  worker  has  a  new  job: 
converting  knowledge  into  actions  that 
convert  information  from  one  form  to 
another. 


Because  the  behaviors  of  a  knowledge 
worker  are  primarily  private,  supervisors 
cannot  supervise. 


Due  do  the  nature  of  knowledge  work,  it  is 
the  worker  that  has  almost  total  authority  in 
matching  methods  to  the  varying  job  tasks 
and  situations  that  they  encounter. 


However,  with  this  reality,  there  is  also  a  shift  in  responsibility  .. 
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Managing  the  Work 


No. 

Management  provides  goals. 
Knowledge  workers  manage  their  work. 


Software  Engineering  Institute 
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Controlling  Your  Own  Destiny 

To  control  the  way  they  work,  software  engineers  must  plan  their 
projects. 

For  management  to  trust  these  plans,  the  engineers  must  make 
accurate  pians. 

To  make  accurate  plans,  they  must  have  data. 

To  have  data,  they  must  measure  their 
performance. 


INTtGkftTiON 


TRANSFORMATION 


ANALYSIS 


\ 

exploration 


LtA/?A/lN& 


You  Need  Data  To  Manage  Yourself 


Metrics  are  good. 


But ... 
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This  Is  A  Data  Collection  Fallacy 


GOOD 


BETTER 

BEST 

Metrics  are  good. 
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Only  Four  Basic  Measures  Needed 


Software  engineers  only  need  to  collect  four  basic  measures  to  mange  their 
schedule  performance  and  the  quality  of  their  work. 


Software  Engineering  Institute 
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Measures  Are  Estimated  and  Then  Tracked 


At  the  beginning  of  an  effort,  the  work  is  planned  and  divided  into  a  set  of  tasks 
or  activities  called  phases.  The  basic  measures  are  estimated. 

•  product  size 

•  time-in-phase 

•  defects  injected  into  a  phase 

•  defects  found  in  a  phase 

•  task  completion  dates 

During  the  project,  these  measures  are  collected  in  real  time 

•  time-in-phase 

•  defect  type  injected  in  phase 

•  find/fix  time  for  each  defect 

•  task  completion  dates 

When  a  product  has  been  completed 

•  product  size  is  measured 
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No  one  wants  to 
be  measured  by 
others. 


r 


=^=-  Software  Engineering  Institute  Gan 


And,  that’s  not  what 
we’re  talking  about 
here. 


de  Mellon  University 
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Tool  Support  For  Data  Collection 


Collecting  the  four  core  measures  would  be  impractical  in  the  absence  of 
software  support. 

A  number  of  tools  are  available  that  make  it  easy  to  collect  this  type  of  data. 
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Collecting  Personal  Data 


I  .../Course  Slidev/ork/3  -  T5P  Process  Elements 

zi 

1  '4^ 

Jl 

Running.  Started  10:56  Ar>1. 

Interrupt  Time  (hhimm): 

- 

rr 

^  ‘  i  w 

Automatically  entered  into  time  log. 


t 

Click  to  enter 
defect  into  log 


^  WBS  ‘  ♦  Tasks  |  Time  Log  |  ^  Defect  Log  Q  Plan  ^  Administration  Q  Reports  .  Inspections  |  ’  Show  Dashboard  | 

ft 

Only  show  entries  from  the  last|  14  days 

Task  Time  Spent  (hh:mm’ 

)  Intemjpt  (hhrmni)  Net  Time  Timestamp 

Contments 

Customers  /  Northrop  Grumman  /  Develop  PM  Report  /  Develop  PM  Report  -  Prod  Dev  -  Report  INSP-1 

2:05 

0:00 

2:05 

11/12/2012  7:09:21  AM 

InvesBgabng  and  responding  (by  en 

Customers  /  Northrup  Grumman  /  Develop  PM  Report  /  Develop  PM  Report  -  Prod  Dev  -  Report  INSP-1 

1:2S 

0:00 

1:25 

11/13/2012  2:13:08  PM 

Adding  diagrams  requested  by  Matt 

#  Customers  /  Northrup  Grumman  /  Develop  PM  Report  /  Develop  PM  Report  -  Prod  Dev  -  Report  INSP-1 

0:32 

0:00 

0:32 

11/14/2012  2:16:36  PM 

Re-inspection  of  doc  after  making  cf 

^  Customers  /  NAVO  /  WPB  /  Revise  Team  Member  Training  /  Revise  Team  Member  Training  -  Prod  Dev  -  Implement 

0:43 

0:00 

0:43 

11/16/2012  6:21:06  AM 

Develop  ideas  ar>d  sketch  exerdses  t 

Customers  /  NAVO  /  WPB  /  Revise  Team  Member  Training  /  Revise  Team  Member  Training  -  Review  with  NAVO  [Nov.  16] 

2:32 

0:00 

2:32 

11/16/2012  4:30:05  PM 

Prep  and  conduct  review  walkthroug 

Products  /  Leading  a  Development  Team  -  Revision  /  Leading  a  Development  Team  -  Walcthrough 

6:16 

1:00 

5:16 

11/21/2012  6:23:08  AM 

CorxJuct  walkthrough  vrith  Bill  and  0 

#  Customers  /  Northrup  Grumman  /  Develop  PM  Report  /  Develop  PM  Report  -  Prod  Dev  -  Report  INSP-1 

0:45 

0:00 

0:45 

11/21/2012  6:26:42  AM 

Released  final  version  to  team;  Resp 

#  Customers  /  Northrup  Grumman  /  Develop  PM  Report  /  Develop  PM  Report  -  Prod  Dev  -  Report  INSP-1 

0:53 

0:00 

0:53 

11/21/2012  6:29:51  AM 

Responded  to  several  emails  from  ts 

Products  /  Leading  a  Development  Team  -  Revision  /  Siideware  and  exercises  /  Process  Nov.  19  walcthrough  comments,  prioritize  and  replan 

0:49 

0:00 

0:49 

11/26/2012  11:44:49  AM 

#  Products  /  Leading  a  Development  Team  -  Revision  /  Siideware  and  exercises  /  Process  Nov.  19  walcthrough  comments,  prioritize  and  replan 

0:43 

0:00 

0:43 

11/27/2012  2:17:11  PM 

•  Customers  /  Northrup  Grumman  /  Develop  PM  Report  /  Develop  PM  Report  -  Prod  Dev  -  Release 

1:30 

0:00 

1:30 

11/28/2012  11:31:21  AM 

Make  final  changes  and  corxJuct  fina 

•  Products  /  Leading  a  Development  Team  -  Revision  /  Siideware  aixl  exercises  /  Implement  change  requests  from  Walk-through  -  Globals  and  Mod  1-2 

2:36 

0:00 

2:36 

11/28/2012  3:00:48  PM 

Products  /  Leading  a  Development  Team  -  Revision  /  Siideware  and  exercises  /  Implement  change  requests  from  Walk-through  -  Globals  and  Mod  1-2 

0:22 

0:00 

0:22 

11/28/2012  3:46:25  PM 

#  Products  /  Leading  a  Development  Team  -  Revision  /  Siideware  and  exerdses  /  Implement  char>ge  requests  from  Walk-through  -  Globals  and  Mod  1-2 

0:06 

0:00 

0:06 

11/29/2012  6:33:24  AM 

#  Products  /  Leading  a  Development  Team  -  Revision  /  Siideware  and  exercises  /  Implement  char>ge  requests  from  Walk-through  -  Globals  and  Mod  1-2 

1:41 

0:00 

1:41 

11/29/2012  9:22:42  AM 

Products  /  Leading  a  Development  Team  -  Revision  /  Siideware  and  exerdses  /  Implennent  char>ge  requests  from  Walk-through  -  Globals  and  Mod  1-2 

3:51 

18:00 

0:00 

11/29/2012  5:12:41  PM 

#  Products  /  Leading  a  Development  Team  -  Revision  /  Siideware  and  exerdses  /  Implement  chartge  requests  from  Walk-through  -  Globals  and  Mod  1-2 

1:14 

0:00 

1:14 

11/30/2012  9:29:35  AM 

#  Products  /  Leading  a  Development  Team  -  Revision  /  Siideware  arxJ  exercises  /  Implenwrit  change  requests  from  Walk-through  -  Globals  and  Mod  1-2 

2:05 

0:10 

1:55 

11/30/2012  12:18:52  PM 

#  Products  /  Leading  a  Development  Team  -  Revision  /  Siideware  and  exercises  /  Implenoent  change  requests  from  Walk-through  -  Globals  and  Mod  1-2 

2:05 

0:00 

2:05 

11/30/2012  4:05:30  PM 

Customers  /  Northrup  Grumman  /  Develop  PM  Report  /  Develop  PM  Report  -  Prod  Dev  -  Release 

0:36 

0:00 

0:36 

12/2/2012  8:03:09  AM 

Final  :  relabel  document  review  of  p 

#  Products  /  Leading  a  Development  Team  -  Revision  /  Siideware  and  exerdses  /  Implement  change  requests  from  Walk-through  -  Globals  and  Mod  1-2 

1:28 

0:10 

1:18 

12/3/2012  9:41:08  AM 
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Tracking  Your  Time 


All  activities  that  contribute  to  the  value  chain  are  listed  as  tasks  in  your 
plan.  Work  against  any  task  in  your  plan  is  timed. 

•  When  you  begin  work  on  a  task,  you  start  the  timer  in  the  tool. 

•  When  you  stop  work  on  a  task,  you  stop  the  timer. 

•  The  tool  calculates  durations  automatically. 

•  If  your  task  is  interrupted,  you  can  stop  and  then 
restart  your  timer  to  resume. 

•  If  you  forget  to  use  your  timer,  you  estimate 
your  time-on-task,  and  enter  it  manually. 
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How  Am  I  Spending  My  Time? 


Hi.  i’m  Joe. 


Estimate 


Code  Test 
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How  Am  I  Spending  My  Time? 


Wow!  What  happened? 


Estimate 


Code 


Test 


Actual 


Code 


Test 
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How  Am  I  Spending  My  Time? 

 Time 


Before 

Code  Test 

Incorporate  design  before  beginning  code 

After 

Design  Code  Test 

I 

I 

I 

Savings 
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Analysis  of  Data  to  Improve 


80- 

70- 

60- 

Percent 
Defects 
Found  40- 

30- 

20- 

10- 

0- 

0 


100  200  300  400  500  600  700 

Review  Rate  (Lines  of  Code  per  Hour) 


[Humphrey  2005,  page  192] 
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Analysis  of  Data  to  Improve 


80- 

70- 

60- 

Percent 
Defects 
Found  40- 

30- 

20- 

10- 

0- 

0 


Optimal  review  rate  for  this 
SW  Engineer:  Approx.  100  LOC/Hr. 


100  200  300  400  500  600  700 

Review  Rate  (Lines  of  Code  per  Hour) 


[Humphrey  2005,  page  192] 
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What  About  Quality  Performance? 


In  the  absence  of  measurement ... 

9 


Software 

code 


No  feedback  loop 


■  No  learning 


■  Mistakes  will  happen  again 
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Defect  Tracking 


Most  defects  are  discovered  during  personal 
reviews,  inspections,  and  other  quality  control 
activities. 

Whenever  a  defect  is  found,  you  open  the 
defect  log  of  the  planning/tracking  tool  and 
record  the  following: 

•  the  start  time  when  defect  was  found 


•  defect  type 

•  the  process  phase  where  the  defect  was  injected 

•  the  process  phase  where  the  defect  was  removed 

•  a  brief  description  of  the  defect 

•  the  stop  time  (when  you  have  completed  fixing  the  defect) 
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Fix  Time  by  Defect  Type 


400 

350 

300 

250 

Fix  Time 
(Min.)  200 

150  I 

100  - 

50  - 

0  ‘ 


>6° 


nO  re/ 


<,0  ^ 


...r  c/ 


^ 

<#  ^ 


(f 


#  / 

</ 


Defect  Type 
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Improving  Review  Practices 


Percent 

Defects 

Found 


Defect  Type 


update  review 
checklist. 
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from  occurring 
again  and  again. 
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Using  Measurement  To  Understand 


...  and,  to  get  better 


Review  & 
Unit  Test 

%  ^ 

Software 

code  # 


Closed-loop 
feedback  &  learning 


Defects  recorded  by  type 
and  time-to-fix 

Closed-loop  feedback 

Learn  from  mistakes  so  they 
don’t  happen  again 
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Taking  Responsibility 


Others  cannot  manage  how  you 
estimate  your  work  and  how  you 
manage  the  quality  of  our  work. 

Knowledge  workers  manage 
themselves  with  data. 

Software  engineers  are  knowledge 
workers. 

Only  four  basic  measures  are  need 
to  manage  your  work. 


Software  Engineering  Institute 


Carnegie  Mellon  University 


When  Measurement  Benefits  the  Measured 
Kasunic  &  Nichols,  April  23,  2014 

©  2014  Carnegie  Mellon  University 


47 


Can  software  engineers  leverage  goal¬ 
setting  and  measurement  the  way  that 
star  athletes  do? 
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When  Measurement  Benefits  the  Measured 
by  Mark  Kasunic  and  William  Nichols 
04.23.14*  1:30  pm  ET-2:30  pm  ET 


William  Nichols 

Senior  Member  of  the  Teehnieal  Staff 
Software  Engineering  Institute 

Bill  Nichols  joined  the  Software  Engineering  Institute  (SEI)  in  2006  as  a  senior 
member  of  the  technical  staff  and  serves  as  a  Personal  Software  Process  (PSP) 
instructor  and  Team  Software  Process  (TSP)  Mentor  Coach  with  the  TSP  Initiative 
within  the  Software  Solutions  Division  (SSD).  Prior  to  joining  the  SEI,  Dr.  Nichols 
lead  a  software  development  team  at  the  Bettis  Laboratory  near  Pittsburgh, 
Pennsylvania,  where  he  had  been  developing  and  maintaining  nuclear  engineering  and 
scientific  software  for  14  years.  His  TSP  publications  include  the  the  PSP  and  TSP 
Bodies  of  Knowledge,  The  TSP  Coach  Mentoring  Program  Guidebook,  and  various 
publications  addressing  software  quality  planning.  Research  publications  include  an 
algorithm  for  use  in  neutron  diffusion  programs,  design  and  performance  of  a  physics 
data  acquisition  system,  and  experimental  results  in  particle  physics.  He  has  a 
doctorate  in  physics  from  Carnegie  Mellon  University. 
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Measurement  On  Your  Team 


The  Team 
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Self-Managed  Team  of  Knowledge  Workers 


Project  Manager 


Q  Q  Q  Q  Q  Q  Q 

xxxxxxx 


Development  Staff 


Traditional  Project 


The  PM  plans,  directs,  and  tracks  the  work. 


Self-Managed  Team 


The  Team  directs,  and  tracks  the  work. 
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Self-Managed  Teams  Plan  Their  Work 


Management  provides  the  goals  and 
constraints  for  the  project. 

The  team  then  develops  its  plan  for 
meeting  management’s  objectives. 

If  necessary,  the  team  negotiates  with 
management  to  arrive  at  a  mutually 
agreeable  outcome. 

But  it’s  up  to  the  team  to  manage  their  work!  Not  someone  else. 

The  team  must  have  data  to  manage  the  work,  to  meet  their  commitments. 

Measurement  helps  teams  manage  their  commitments. 
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Measurement  To  Benefit  the  Measured 


Personal 

I  i  Unit  Tests 

I  I  Inspection 

System  Test 

Reviews 

:  = 

Software 

Software 

Software 

■  ( 

'Software 

code 

code 

code 

$  % 

code 

Measurement  feedback 
to  the  team 
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Measurement  Used  to  Manage 


Project  and 


Adjustments  are  made 
to  the  plan  based  on 
measurement  feedback. 


The  data  is  used 
throughout  to  assess 
performance. 


1 
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Planning,  Doing,  and  Learning 


The  Project 


^  Pplppcp  1  \ 

^  Pplppep  9  \ 

^  Pplppcp  n  \ 

1  XwlwClOW  1 

Cycle  1  Cycle  2 

[ i 

Cycle  3  Cycle  4  Cycle  5 

[ * * 

\  1  XwlwClOW  #  1 

Cycle  6  Cycle  7  •••  Cycle  n 

_ ^ _ ^ _  _ * _ 

I  \l  \l  ]l  \l  \l  \l  \  I  \ 

•  •• 

Cycle  launch 


Postmortem  or  retrospective 

Cycle  (a.k.a.,  iteration  or  sprint) 

Weekly  objectives 
Two-three  tasks  per  team  member 
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Comparing  Estimates  to  Actuals 


For  both  schedule  and  quality  ... 


Is  the  project  on  track? 
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Comparing  Estimates  to  Actuals 


For  both  schedule  and  quality  ... 


Closed  loop  feedback 


Is  the  project  on  track? 
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Comparing  Estimates  to  Actuals 


For  both  schedule  and  quality  ... 


Closed  loop  feedback 
Learning 


Is  the  project  on  track? 
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Comparing  Estimates  to  Actuals 


For  both  schedule  and  quality  ... 


Closed  loop  feedback 
Learning 

Performance  improvement 


Is  the  project  on  track? 
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Measurement  That  Benefits  the  Measured 


Postmortem 
(Retrospective) 
Lessons  Learned 


Team  Planning 
or  Replanning 


Cycle 


Execute 

Measure 


L 


Weekly 

Review 
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Again  ...  Only  Four  Basic  Measures  Needed 


Software  engineers  only  need  to  collect  four  basic  measures  to  mange  their 
schedule  performance  and  the  quality  of  their  work. 
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Derived  From  the  Four  Basic  Measures 


Many  other  useful  measures  and  indicators  can  be  easily  derived  from 
the  four  basic  measures  including: 


estimation  accuracy* 
prediction  intervals* 
time  in  phase  distribution 
defect  injection  distribution 
defect  removal  distribution 
productivity 
reuse  percentage 


cost  performance  index 

planned  value 

earned  value 

predicted  earned  value 

defect  density 

defect  density  by  phase 

defect  removal  rate  by  phase 


defect  removal  leverage 
review  rates 
process  yield 
phase  yield 

failure  cost  of  quality  (COQ) 
appraisal  COQ 
appraisal/failure  COQ  ratio 


*  Both  size  and  time 
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How  Do  You  Know  If  It’s  a  Best  Practice? 


Organizations  want  a  way  to  gauge  their  performance  and  to  compare  their 
performance  with  others  in  their  industry. 

Data  on  project  performance  is  needed  to  provide  evidence  of  what  (exactly) 
constitutes  a  best  practice. 

How  do  you  even  know  what  a  best  practice  is  unless  you  measure  and 
compare  it  to  other  practices? 


Benchmarks  provide  a  reference 
point  for  interpreting  performance. 
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Benchmarking  &  Best  Practices 


Repository 


You  can’t  benchmark  without 
measurement! 
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Description  of  the  Data 


The  data  was  submitted  to  the  SEI  between  2000  and  2012. 

The  source  data  is  from  93  projects  in  the  United  States  and  20 
projects  from  Mexico. 

This  is  data  that  has  been  aggregated  at  the  team  level  at  the 
time  of  a  cycle  postmortem  (retrospective). 

Only  data  from  a  project’s  last  postmortem  is  included. 

Tests  were  conducted  to  ensure  that  extracted  data  represented 
unique  projects. 
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How  Long  Were  Project  Durations?  [Weeks] 


Duration  (Weeks) 
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What  were  Project  Durations?  [Weeks] 


Duration  (Weeks) 
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What  were  Project  Durations?  [Weeks] 


Duration  (Weeks) 
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What  were  Project  Durations?  [Weeks] 


Duration  (Weeks) 
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What  were  Project  Durations?  [Weeks] 


30  H 


25- 


20- 


Frequency  is- 


10- 


5- 


0 


0 


Most  Common  9-12  weeks 
Half  Shorter  than  1 3  weeks 


Mean 

=  16.9 

Median 

=  13.0 

n 

=  113 

Many  a  half  year  or  more 
Why  longer? 


■p. 


16  32  48  64 

Duration  (Weeks) 


80 
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Size  -  Actual  Added  and  Modified 

Thousand  Source  Lines  of  Code  [KLOC] 


60  H 


50- 


40- 


Frequency  30- 


20- 


10- 


Mean  = 
Median  = 
n  = 


11.7 

6.6 

112 


Projects  come  in  a  variety  of  sizes 


I  I 


0  10  20  30  40  50  60 

Actual  Added  &  Modified  Code  (KLOC) 
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Size  -  Actual  Added  and  Modified 

Thousand  Source  Lines  of  Code  [KLOC] 


60  H 


50- 


40- 


Frequency  30- 


20- 


10- 


Mean  = 
Median  = 
n  = 


11.7 

6.6 

112 


Projects  come  in  a  variety  of  sizes 


I  I 


0  10  20  30  40  50  60 

Actual  Added  &  Modified  Code  (KLOC) 
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Size  -  Actual  Added  and  Modified 

Thousand  Source  Lines  of  Code  [KLOC] 


60  H 


50- 


40- 


Frequency  30- 


20- 


10- 


Mean  = 
Median  = 
n  = 


11.7 

6.6 

112 


But  how  much  code  a  team  can  write 


I 


0  10  20  30  40  50  60 

Actual  Added  &  Modified  Code  (KLOC) 
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Team  Size 


Frequency 


=^=-  Software  Engineering  Institute  Gan 


Mean  =  8.2 
Median  =  8.0 
n  =  111 


But  how  much  code  a  team  can  write 
also  depends  upon  team  size. 


Team  Size 
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Team  Size 


Frequency 


=^=-  Software  Engineering  Institute  Gan 


Mean  =  8.2 
Median  =  8.0 
n  =  111 


Are  these  larger  teams  productive? 


Team  Size 
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Team  Size 


Frequency 


=^=-  Software  Engineering  Institute  Gan 


Mean  =  8.2 
Median  =  8.0 
n  =  111 


Are  these  larger  teams  productive? 


How  would  we  measure? 


9  12  15  18 

Team  Size 
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Code  Production  Rate 


Productivity  (LOC/Hr) 
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Team  Size  vs.  Productivity 
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100  n 


Productivity 

(LOC/Hr) 


0 


=  111 


Overall,  weak  relationship 


Slope  is  not  a  useful  predictor 


10 

Team  Size 


15 


20 
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What  do  you  think  is  the  average  number  of  task  hours  that  a  team 
member  spends  during  a  week? 


□  35  to  40  hours 

□  30  to  35  hours 

□  25  to  30  hours 

□  20  to  25  hours 

□  15  to  20  hours 

□  less  than  15  hours 
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Mean  Team  Member  Weekly  Task  Hours 


Frequency 


Mean  Team  Member  Weekly  Task  Hours 
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Mean  Team  Member  Weekly  Task  Hours 


Frequency 


Mean  Team  Member  Weekly  Task  Hours 
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Plan  Vs.  Actual  Hours 


Actual  Hours  ‘ 
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Plan  Vs.  Actual  Hours 


Actual  Hours  ‘ 


n  =  113 
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Plan  Vs.  Actual  Hours 


Actual  Hours  ‘ 


n  =  113 
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Let’s  Looks  at  Some 
Quality-Based  Profiles 
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Total  Defects  Injected  Per  KLOC 


Frequency 


Defects  Injected  per  KLOC 
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Total  Defects  Injected  Per  KLOC 


To  terror  is  human. 


Frequency 


Defects  Injected  per  KLOC 
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Total  Defects  Injected  Per  KLOC 


To  terror  is  human. 


Frequency 


Defects  Injected  per  KLOC 
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Total  Defects  Injected  Per  KLOC 


To  terror  is  human. 


Frequency 


Defects  Injected  per  KLOC 
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Injection  and  Removal  of  Defects 


Requirements,  design,  code 


Defect  Injection  phase 


Intermediate  product  with  defects 

it _ 


Defect  removal  phase 


Phase  Yield  Reviews,  inspections,  tests 

(%  defects  removed) 
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Multiple  Defect  Removal  Filters  Required 


Multiple  defect  removal 
filters  are  required  to  realize 
high-quality  software. 

■  Personal  reviews 

■  Inspections 

■  Tests 
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Defect  Density  -  Summary 


Frequency 


Defects  Per  KLOC  -  Code  Review 


Frequency 


Frequency 


Defects  Per  KLOC  -  Unit  Test 


Frequency 


Defects  Per  KLOC  -  Build  and  Integration  1 


Frequency 


Defects  Per  KLOC  -  System  Test 
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Defect  Density  -  Summary 


Frequency 


Defects  Per  KLOC  -  DLD  Review  Defects  Per  KLOC  -  Code  Review 


Frequency 


Frequency 


Defects  Per  KLOC  -  Unit  Test 


Frequency 


Defects  Per  KLOC  -  Build  and  Integration  1 


Frequency 


Defects  Per  KLOC  -  System  Test 
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Defect  Density  -  Summary 


Frequency 


Defects  Per  KLOC  -  DLD  Review  Defects  Per  KLOC  -  Code  Review 


Defects  Per  KLOC  -  Inspection 


Frequency 


Defects  Per  KLOC  -  Unit  Test 


Frequency 


Defects  Per  KLOC  -  Build  and  Integration  1 


Frequency 


Defects  Per  KLOC  -  System  Test 


— ^  When  Measurement  Benefits  the  Measured 

Software  Engineering  Institute  Carnegie  Mellon  University  Kasunics  Nichols,  April  23, 2014  98 

©  2014  Carnegie  Mellon  University 


Defect  Density  -  Summary 


Frequency 


Defects  Per  KLOC  -  DLD  Review  Defects  Per  KLOC  -  Code  Review 


Defects  Per  KLOC  -  Inspection 


Frequency 


Test  Removes  Other  Defects 


Defects  Per  KLOC  -  Unit  Test 


Frequency 


Defects  Per  KLOC  -  Build  and  Integration  1 


Frequency 


Defects  Per  KLOC  -  System  Test 


— ^  When  Measurement  Benefits  the  Measured 

Software  Engineering  Institute  Carnegie  Mellon  University  Kasunics  Nichols,  April  23, 2014  99 

©  2014  Carnegie  Mellon  University 


Defect  Removal  Density  - 

Median  of  Defects  Per  KLOC 


DLD  Review 


Code  Review 


Code  Inspection 


Unit  Test 


Build/Integration  Test 


System  Test 


How  do  you  know  if  you  are  on  track? 


<5 


<0.5 


I  0.15 


0.2 


0  1  2  3  4  5  6 


Defects  Per  KLOC  (Median) 
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^  DOWNLOA^ 


All  project  performance  charts 
are  available  as  a  download  with 
today’s  webinar. 

These  include  charts  not 
presented  in  this  webinar. 
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W.  Edwards  Deming 


In  God  we  trust ... 

All  others  bring  uiDatad 
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Contact  Information 


Mark  Kasunic 

Senior  Member  of  Technical  Staff 
Software  Engineering  &  Acquisition  Practices 
Software  Solutions  Division 
Telephone:  +1  412-268-5863 
Email:  info@sei.cmu.edu 


U.S.  Mail 

Software  Engineering  Institute 
Customer  Relations 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-2612 
USA 


Web 

www.sei.cmu.edu 

www.sei.cmu.edu/contact.cfm 


Customer  Relations 

Email:  info@sei.cmu.edu 
Telephone:  +1  412-268-5800 

SEI  Phone:  +1  412-268-5800 

SEI  Fax:  +1  412-268-6257 
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